feat: mutiple backends for recipes #3114
Conversation
Fixes containers#3006 Signed-off-by: Jeff MAURY <[email protected]>
Fixes containers#3006 Signed-off-by: Jeff MAURY <[email protected]>
| basedir: 'recipes/natural_language_processing/chatbot', | ||
| readme: '', | ||
| recommended: ['hf.instructlab.granite-7b-lab-GGUF', 'hf.instructlab.merlinite-7b-lab-GGUF'], | ||
| backend: 'llama-cpp', |
There was a problem hiding this comment.
shouldnt this be also backends: ['llama-cpp'] ?
There was a problem hiding this comment.
Not mandatory because it will be parsed but updated for consistency
packages/backend/src/assets/ai.json
Outdated
| @@ -166,7 +166,7 @@ | |||
| "hf.instructlab.merlinite-7b-lab-GGUF", | |||
| "hf.lmstudio-community.granite-3.0-8b-instruct-GGUF" | |||
| ], | |||
| "backend": "llama-cpp", | |||
| "backend": ["llama-cpp", "openvino"], | |||
There was a problem hiding this comment.
Shouldnt there be also backends like if I have an array -> backedns
Signed-off-by: Jeff MAURY <[email protected]>
| .flatMap(r => r.backends) | ||
| .filter(b => b !== undefined) | ||
| .filter((value, index, array) => array.indexOf(value) === index) | ||
| .sort((a, b) => a.localeCompare(b)) |
There was a problem hiding this comment.
remark (non-blocking): non related to this PR, but this filtering chain is unreadable, what is r ? what is b?, what is the type of value, what is the type of a and b? :/
| let res: Recipe[] = []; | ||
| for (const value of values) { | ||
| res = [...res, ...result.filter(r => r.backends?.includes(value))]; | ||
| } | ||
| result = res; |
There was a problem hiding this comment.
question: I don't understand what this code is doing? can we add a comment? filtering result? shallow copy?
| @@ -42,7 +42,7 @@ let startedContainerProviderConnectionInfo: ContainerProviderConnectionInfo[] = | |||
| let localPath: LocalRepository | undefined = $derived(findLocalRepositoryByRecipeId($localRepositories, recipe?.id)); | |||
| // Filter all models based on backend property | |||
| let models: ModelInfo[] = $derived( | |||
| $modelsInfo.filter(model => (model.backend ?? InferenceType.NONE) === (recipe?.backend ?? InferenceType.NONE)), | |||
| $modelsInfo.filter(model => recipe?.backends?.includes(model.backend ?? InferenceType.NONE)), | |||
There was a problem hiding this comment.
suggestion: I would be more confortable adding an unit test in StartRecipe.spec.ts where we have a recipe with several backends, to ensure we properly display all models matching the backends listed.
| @@ -42,7 +42,7 @@ let startedContainerProviderConnectionInfo: ContainerProviderConnectionInfo[] = | |||
| let localPath: LocalRepository | undefined = $derived(findLocalRepositoryByRecipeId($localRepositories, recipe?.id)); | |||
| // Filter all models based on backend property | |||
Signed-off-by: Jeff MAURY <[email protected]>
Fixes #3006
Signed-off-by: Jeff MAURY [email protected]
What does this PR do?
Allow association from recipe to multiple backends
Screenshot / video of UI
N/A
What issues does this PR fix or reference?
#3006
How to test this PR?
Start recipes. You should be a able to start recipes with an OpenVINO model